setup_docker: make LXC resource limits visible to nested containers - #9
setup_docker: make LXC resource limits visible to nested containers#9andrebrait wants to merge 1 commit into
Conversation
Try this branchThe engine and the scripts resolve independently, so a production script can COMMUNITY_SCRIPTS_CORE_URL=https://raw.githubusercontent.com/andrebrait/core/fix/docker-lxc-resource-visibility \
bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVED/main/ct/debian.sh)"Swap Run a script from a fork as wellcurl -fsSL https://raw.githubusercontent.com/andrebrait/core/fix/docker-lxc-resource-visibility/tools/run.sh |
bash -s -- https://raw.githubusercontent.com/YOU/ProxmoxVED/your-branch ct/debian.sh \
https://raw.githubusercontent.com/andrebrait/core/fix/docker-lxc-resource-visibilityNote that Useful flags while testing
|
Inside an LXC container, Docker gives every container it starts a fresh procfs in its own mount namespace. The lxcfs bind mounts that LXC placed on /proc/meminfo and friends live in the CT's namespace and never reach it, so each Docker container reads the physical host's totals instead of the CT's limits: CT sees: MemTotal: 8388608 kB container sees: MemTotal: 65648168 kB Enforcement was never affected, since the CT cgroup still caps every descendant. Visibility was, and anything that self-sizes from those numbers - JVM heap ergonomics, Node, Go, OpenMP thread pools - sizes against the wrong value and gets OOM-killed instead of throttling. Register a small runc wrapper as Docker's default runtime. On each container create it re-binds the lxcfs-backed files into the container and, when the container sets no limit of its own, applies the CT's memory limit to the container cgroup. Both halves are needed because runtimes read the limit two different ways: file readers use /proc, while musl and Node's uv_get_constrained_memory() consult the cgroup. CPU needs no handling - cpuset is inherited down the cgroup tree, so nested containers already observe the CT's core count. The one exception is /sys/devices/system/cpu/online, which glibc's sysconf(_SC_NPROCESSORS_ONLN) reads, so it is bound alongside the /proc files. The helper is a no-op outside an LXC container and wherever lxcfs is not mounted, so the five non-Docker installers that call setup_docker are unaffected. It fails open: no runc, no jq, or malformed JSON all fall through to the real runc unmodified, degrading to current behaviour rather than breaking the daemon. Opt out with DOCKER_LXCFS_VISIBILITY=false. Regenerate lib/API.txt for the new function.
b644b18 to
c2dbd48
Compare
|
Adding a caveat found in live testing after opening this, now documented in the function header. Monitoring containers are the one case that wants the opposite of this patch. lxcfs resolves usage by the reading process's own cgroup. Once Two things worth noting about the scope of this:
Tools that read the Docker API rather than Escape hatch is per container, no config change needed elsewhere: services:
arcane:
runtime: runcI don't think this should change the default. The patch fixes every workload that self-sizes from the wrong number, which is the common case; monitoring containers are a small, identifiable set with a one-line opt-out. But it is a real behaviour change for them and maintainers should weigh it — happy to gate it differently if you'd prefer, e.g. skipping containers that mount |
✍️ Description
Inside an LXC container, Docker gives every container it starts a fresh procfs in its own mount namespace. The lxcfs bind mounts that LXC placed on
/proc/meminfoand friends live in the CT's namespace and never reach it, so every Docker container reads the physical host's totals instead of the CT's limits:Enforcement was never affected — the CT cgroup still caps every descendant. Visibility was, and anything that self-sizes from those numbers (JVM heap ergonomics, Node, Go, OpenMP thread pools) sizes against the wrong value and gets OOM-killed instead of throttling.
This registers a small runc wrapper as Docker's
default-runtime. On each container create it re-binds the lxcfs-backed files into the container and, when the container sets no limit of its own, applies the CT's memory limit to the container cgroup.Both halves are needed, because runtimes read the limit two different ways:
sysconf(_SC_PHYS_PAGES), procpsfreeand friends read/proc/meminfo→ fixed by the binds.uv_get_constrained_memory()consult the cgroup → fixed by the limit.CPU needs no handling:
cpusetis inherited down the cgroup tree, so nested containers already observe the CT's core count. The one exception is/sys/devices/system/cpu/online, which glibc'ssysconf(_SC_NPROCESSORS_ONLN)reads, so it is bound alongside the/procfiles. Without it, a container on a 2-core CT reportsgetconf _NPROCESSORS_ONLN= 6 on a 6-core host whilenproccorrectly says 2 (coreutils usessched_getaffinity).Toggle
DOCKER_LXCFS_VISIBILITY, defaulttrue, documented in thesetup_docker()header alongside the existingUSE_DOCKER_REPO/DOCKER_LOG_DRIVER/DOCKER_SKIP_UPDATES.lxc-attachinherits the environment, so users opt out with:DOCKER_LXCFS_VISIBILITY=false bash -c "$(curl -fsSL .../ct/docker.sh)"Safety
grep -qs 'fuse\.lxcfs' /proc/mounts), so the five non-Docker installers that callsetup_dockerare unaffected.runc, nojq, malformed JSON, or empty output all fall through to the real runc unmodified — worst case is current behaviour, never a dead daemon.runcresolved at runtime, since distrodocker.ioships/usr/sbin/runcwhilecontainerd.ioships/usr/bin/runc.--memory/mem_limit.docker run --runtime=runcbypasses the wrapper for a single container.systemctl enable -q --now docker, so the daemon starts with the runtime already registered — no restart needed.jqmerge preserves any existingdaemon.jsonkeys (registry-mirrors,insecure-registries,log-driver).Known limitation, documented in the function header
The limit is written at container creation. After changing a CT's memory with
pct set, existing containers keep the old value until recreated, or re-synced withdocker update --memory=<new> --memory-swap=<new> <container>. New containers pick up the new value immediately, and CPU changes propagate live with no action.The alternative — binds only, no cgroup limit — tracks resizes live (lxcfs walks up to the nearest ancestor with a limit) but leaves Node and musl reading the host. I judged that the worse default; happy to flip it if maintainers disagree.
🔗 Related Issue
Fixes community-scripts/ProxmoxVE#16756
✅ Prerequisites (X in brackets)
API.txtwas updated to match.🤖 AI Assistance (X in brackets)
Model: Claude Opus 5 (1M context), reasoning effort high, via Claude Code.
🛠️ Type of Change (X in brackets)
README,AppName.md,CONTRIBUTING.md, or other docs.Test evidence
Run against a real Proxmox VE 9.2.10 host (PVE kernel 7.0.14-11-pve, lxcfs 7.0.0-pve1) in an unprivileged Debian 13 CT with
features: nesting=1, 8192 MB / 6 cores, Docker 29.7.2, on a 62 GiB host. The CT was reset to a clean pre-patch state, then the patched_docker_setup_lxcfs_visibilitywas extracted fromlib/runtime.funcand executed:Additional cases, same CT:
Note
[B]: lxcfs resolves per reading process's cgroup, so a container with its own--memoryreports its limit through the same bind — no per-container wiring needed.CI gates run locally:
Verification one-liner for reviewers
Two commands that look fine while this is broken, worth avoiding:
docker inforeports the correctMemTotalthroughout, because the daemon is a plain CT process reading lxcfs; anddocker run --rm alpine free -mprints the host figure even once fixed, because busyboxfreeusessysinfo(2), which lxcfs cannot intercept.